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SEP -2  2005 

MEMORANDUM  FOR  AUDITOR  GENERAL,  DEPARTMENT  OF  THE  NAVY 

SUBJECT:  Quality  Control  Review  of  Naval  Audit  Service’s  Special  Access  Program 
Audits  (Report  No.  D-2005-6-010) 

We  are  providing  this  report  for  your  review  and  comment.  We  have  reviewed  the 
Naval  Audit  Service  (NAVAUDSVC)  system  of  quality  control  used  on  Special  Access 
Program  (SAP)  audits  for  the  three  years  ended  September  30,  2004.  The  Government 
Auditing  Standards  (GAS)  require  that  an  audit  organization  performing  audits  and/or 
attestation  engagements  in  accordance  with  GAS  should  have  an  appropriate  internal 
quality  control  system  in  place  and  undergo  an  external  peer  review  at  least  once  every 
3  years  by  reviewers  independent  of  the  audit  organization  being  reviewed.  As  the 
organization  that  has  audit  policy  and  oversight  responsibilities  for  audits  in  the 
Department  of  Defense,  we  conducted  this  external  peer  review  of  the  NAVAUDSVC 
audits  requiring  special  access  in  conjunction  with  the  Army  Audit  Agency’s  external 
peer  review  of  NAVAUDSVC  non-SAP  audits. 

An  audit  organization’s  quality  control  policies  and  procedures  should  be 
appropriately  comprehensive  and  suitably  designed  to  provide  reasonable  assurance  of 
meeting  the  objectives  of  quality  control.  We  tested  the  NAVAUDSVC  SAP  system  of 
quality  control  to  the  extent  considered  appropriate. 

In  our  opinion,  the  NAVAUDSVC  system  of  quality  control  used  on  SAP  audits 
in  effect  for  the  period  ended  September  30, 2004,  was  designed  in  accordance  with 
quality  standards  established  by  GAS.  Further,  the  internal  quality  control  system  was 
operating  effectively  to  provide  reasonable  assurance  that  SAP  audit  personnel  were 
following  established  policies,  procedures,  and  applicable  auditing  standards. 
Accordingly,  we  are  issuing  an  unqualified  opinion  on  your  quality  control  system  used 
on  SAP  audits  for  the  review  period  ended  September  30,  2004. 

Appendix  A  contains  the  scope  and  methodology  of  the  review.  Appendix  B 
contains  comments,  observations,  and  recommendations  where  NAVAUDSVC  can 
improve  its  quality  control  program  related  to  SAP  audits,  as  well  as  our  responses  to 
NAVAUDSVC  management  comments  to  the  draft  report.  Appendix  C  provides  the  full 
text  of  management  comments  in  response  to  the  draft  report.  Please  provide  additional 
comments  in  response  to  Appendix  B  by  September  19,  2005. 


We  wish  to  express  our  thanks  to  you  and  your  staff  for  your  cooperation  and 
professionalism.  Please  contact  Mr.  Robert  L.  Kienitz  at  (703)  604-8754 
(DSN  664-8754)  or  Ms.  Carolyn  R.  Davis  at  (703)  604-8877  (DSN  664-8877)  if  you 
have  any  questions. 


JatriciaA.  Brannin 
Assistant  Inspector  General 
Audit  Policy  and  Oversight 


Cc:  Director,  DoD  Special  Access  Program  Coordination  Office 
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Appendix  A.  Scope  and  Methodology 


We  limited  our  review  to  the  adequacy  of  NAVAUDSVC  SAP  auditors’  compliance  with 
quality  policies,  procedures,  and  standards.  We  judgmentally  selected  3  SAP  audits  from 
a  universe  of  1 1  formal  reports  requiring  special  access  issued  by  the  NAVAUDSVC 
Special  Audits  Division  in  FYs  2002,  2003,  and  2004,  and  tested  each  audit  for 
compliance  with  the  NAVAUDSVC  system  of  quality  control.  The  Army  Audit  Agency 
(AAA)  conducted  a  review  of  the  NAVAUDSVC  internal  quality  control  system  for  non- 
SAP  audits  and/or  attestation  engagements  and  has  issued  a  separate  report.  The 
Assistant  Inspector  General  for  Audit  Policy  and  Oversight  will  issue  an  overall  opinion 
report  on  the  NAVAUDSVC  internal  quality  control  system  that  will  include  the 
combined  results  of  the  reviews  of  SAP  and  non-SAP  audits. 

In  performing  our  review,  we  considered  the  requirements  of  quality  control  standards 
and  other  auditing  standards  contained  in  the  2003  Revision  of  the  Government  Auditing 
Standards  (GAS)  issued  by  the  Comptroller  General  of  the  United  States.  GAS  3.52 
states: 


The  external  peer  review  should  determine  whether,  during  the  period  under  review,  the 
reviewed  audit  organization’s  internal  quality  control  system  was  adequate  and  whether 
quality  control  policies  and  procedures  were  being  complied  with  to  provide  the  audit 
organization  with  reasonable  assurance  of  conforming  with  applicable  professional 
standards.  Audit  organizations  should  take  remedial,  corrective  actions  based  on  the 
results  of  the  peer  review. 

We  conducted  this  review  in  accordance  with  standards  and  guidelines  established  in  the 
Draft  2004  President’s  Council  on  Integrity  and  Efficiency  (PCIE)  “Guide  for 
Conducting  External  Peer  Reviews  of  the  Audit  Operations  of  Offices  of  Inspector 
General.”  We  modified  the  Guide  to  ensure  consistency  with  the  AAA  review  of 
non-SAP  audits,  and  to  reflect  the  unique  nature  of  auditing  within  a  SAP  environment. 
We  interviewed  NAVAUDSVC  auditors  and  their  managers,  reviewed  NAVAUDSVC 
internal  audit-related  policies  and  procedures.  We  performed  this  review  in  May  through 
June  2005  at  the  NAVAUDSVC  Special  Audits  Division. 

We  used  the  following  criteria  to  select  the  audits  under  review: 

•  Worked  backward  starting  with  FY  2004  audits  in  order  to  review  the  most 
current  quality  assurance  procedures  in  place. 

•  Eliminated  Base  Realignment  and  Closure  audits  because  they  are  not 
considered  typical  audits. 

•  Avoided  audits  with  multiple  SAPs  associated  with  the  audit  for  ease  of  access. 

•  Avoided  audits  that  have  the  same  or  similar  titles  to  ensure  review  of  multiple 
types  of  projects. 
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The  following  table  identifies  the  specific  reports  reviewed. 


Report  Number 

Date 

Title 

N2003-0046 

April  30,  2003 

“Travel  Cards” 

N2003-0013 

November  12,  2002 

“Purchase  Cards” 

N2002-0076 

September  27,  2002 

“Contract  Closeout” 

Limitations  of  Review.  Our  review  would  not  necessarily  disclose  all  weaknesses  in  the 
system  of  quality  control  or  all  instances  of  noncompliance  with  it  because  we  based  our 
review  on  selective  tests.  There  are  inherent  limitations  in  considering  the  potential 
effectiveness  of  any  quality  control  system.  In  performing  most  control  procedures, 
departures  can  result  from  misunderstanding  of  instructions,  mistakes  of  judgment, 
carelessness,  or  other  human  factors.  Projecting  any  evaluation  of  a  quality  control 
system  into  the  future  is  subject  to  the  risk  that  one  or  more  procedures  may  become 
inadequate  because  conditions  may  change  or  the  degree  of  compliance  with  procedures 
may  deteriorate. 
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Appendix  B.  Comments,  Observations,  and 

Recommendations 


We  are  issuing  an  unqualified  opinion  on  this  external  peer  review  because  the  concerns 
we  identified  during  our  review  were  not  cumulatively  significant  to  the  reports’  findings, 
conclusions,  or  recommendations.  Overall,  we  found  that  NAVAUDSVC  could  improve 
the  quality  control  program  and  guidance  for  audits  related  to  the  areas  of  Audit 
Planning,  Supervision,  Evidence  and  Audit  Documentation,  and  Quality  Assurance.  We 
also  noted  one  other  matter  of  interest  related  to  a  PCIE-required  certification  of  working 
paper  completion.  Implementing  the  recommendations  identified  below  would  improve 
the  quality  control  system  and  help  maintain  an  unqualified  opinion. 

Audit  Planning.  GAS  7.02  states  that  “work  is  to  be  adequately  planned,”  and  GAS 
7.07  states  that  “planning  should  be  documented.”  GAS  7.41  requires  auditors  to 
document  the  planning,  and  states  “the  form  and  content  of  the  written  audit  plan  will 
vary  among  audits,  but  should  include  an  audit  program  or  project  plan,  a  memorandum, 
or  other  appropriate  documentation  of  key  decisions  about  the  audit  objectives,  scope, 
and  methodology  and  of  the  auditors’  basis  for  those  decisions.  It  should  be  updated,  as 
necessary,  to  reflect  any  significant  changes  to  the  plan  made  during  the  audit.” 

The  September  2002  NAVAUDSVC  Handbook  Sections  415.2  and  415.3  required  the 
responsible  Audit  Director  and  Assistant  Auditor  General  to  approve  audit  programs 
before  the  start  of  the  audit  verification  phase,  and  to  be  aware  of  any  significant  changes 
to  the  audit  program.  NAVAUDSVC  Handbook  Section  510.5  stated  that  when 
reviewing  individual  working  papers  (including  the  audit  program),  the  Project  Manager 
must  include  his/her  initials  or  signature  and  the  date  of  the  review  on  the  working  papers 
as  evidence  of  the  review.  NAVAUDSVC  Handbook  Section  417  required  auditors  to 
cross-reference  audit  steps  to  supporting  working  papers,  and  NAVAUDSVC  Handbook 
Section  412.1  required  auditors  to  evaluate  the  reliability  of  computer-based  products  to 
determine  risks  in  using  such  products  that  are  significant  to  the  audit  objective(s)  and 
scope. 

We  found  that  NAVAUDSVC  adequately  planned  the  three  projects  we  reviewed; 
however,  improvements  in  documentation  were  needed  for  all  three  projects.  In  one 
project,  the  auditors  created  an  audit  program  that  set  forth  the  objectives  of  the  audit  and 
included  steps  to  address  each  of  the  objectives,  and  the  Project  Manager  documented 
approval  with  initials  and  date.  The  Project  Manager  also  initialed  the  final 
cross-referenced  version  of  the  audit  program.  However,  the  Project  Manager  did  not 
date  his  initials  on  the  final  cross-referenced  version.  In  addition,  the  audit  program  did 
not  include  steps  to  verify  data  received  through  management  from  outside  sources  (the 
travel  card  contractor).  The  Project  Manager  stated  that  although  these  steps  were  not 
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specifically  in  the  audit  program,  he  believed  the  auditors  did  verify  this  data  through 
other  steps  in  the  program. 

In  the  second  project,  we  found  evidence  of  supervisory  involvement  and  approval  of  the 
audit  plan.  However,  the  audit  plan  was  not  cross-referenced  to  the  working  papers. 
Therefore,  we  could  not  tell  whether  all  steps  in  the  audit  plan  were  completed  or 
modified  and  if  so  whether  the  NAVAUDSVC  management  approved  the  modifications. 

In  the  third  project,  we  did  not  find  documented  supervisory  approval  of  the  audit 
program.  Though  we  could  not  tell  in  reviewing  the  audit  program  whether  any 
modifications  were  made  to  the  audit  plan,  this  project  was  the  fifth  in  a  series  of  six,  and 
NAVAUDSVC  auditors  indicated  during  our  review  that  the  plan  had  been  standardized 
by  the  time  this  audit  was  performed.  In  addition,  though  the  audit  program  was 
cross-referenced  to  the  working  papers,  the  program  provided  for  auditors’  initials  and 
date  for  each  step  but  that  column  was  not  completed. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  all  SAP  managers  to  document  their  approval  for  all  original  and  updated 
audit  programs  (including  the  cross-referenced  version),  ensure  that  the  audit  programs 
include  steps  for  verifying  data  obtained  from  outside  sources  during  the  audit,  ensure  the 
audit  program  is  adequately  cross-referenced  to  working  papers,  and  complete  the  initial 
and  date  column  if  such  a  column  is  included  in  the  audit  program. 

Management  Comments.  The  NAVAUDSVC  concurred  with  the 
recommendation  and  stated  that  an  all-hands  e-mail  will  be  issued  by  November  30, 

2005,  reminding  all  personnel  of  existing  NAVAUDSVC  requirements  pertaining  to 
audit  programs  and  evaluating  computer-based  products  to  determine  risks. 

Reviewer  Response.  Management  comments  are  responsive. 

Supervision.  Paragraph  6.22  of  the  June  1994  version  of  GAS  and  paragraph  7.44  of  the 
June  2003  version  of  GAS  state  that  “staff  are  to  be  properly  supervised.”  The 
November  2001  version  of  the  NAVAUDSVC  Handbook  Paragraph  510(4)  stated  that 
the  Project  Manager  or  a  senior  experienced  auditor  should  review  individual  working 
papers.  To  provide  evidence  of  the  review,  the  reviewer  should  include  their  initials  or 
signature  and  the  date  of  the  review  on  the  working  papers.  Also,  the  Handbook  stated 
that  the  Audit  Director  should  review  working  papers  supporting  unusually  sensitive 
findings  and  conclusions  and  that  the  Audit  Director  should  place  emphasis  on  work 
performed  by  the  Project  Manager. 

The  projects  showed  evidence  of  supervisory  involvement  and  oversight;  however, 
improvements  could  be  made  for  documenting  supervisory  review  in  two  projects.  In 
one  project,  while  there  was  evidence  of  supervisory  involvement  which  was  indicated  by 
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the  Project  Manager  signing  the  working  papers  reviewed,  the  Project  Manager  did  not 
provide  a  date  of  the  review  for  7  of  the  10  working  papers  reviewed.  For  the  working 
papers  that  were  dated  by  the  Project  Manager  the  review  was  timely.  However,  for  the 
remaining  working  papers  we  could  not  tell  whether  the  supervisory  review  was  timely. 

In  addition,  none  of  the  12  working  papers  prepared  by  the  Project  Manager  were 
reviewed.  The  Project  Manager  stated  that  the  Audit  Director  relied  on  the  experience 
and  expertise  of  the  Project  Manager. 

In  the  second  project,  while  there  was  evidence  of  supervisory  involvement,  only 
24  percent  of  the  working  papers  we  looked  at  were  reviewed  by  the  supervisor.  We 
reviewed  at  least  7  of  1 1  file  folders  of  working  papers.  We  reviewed  a  total  of 
59  working  papers  and  found  that  only  14  of  the  59  were  reviewed  by  a  supervisor.  Nine 
of  the  59  had  working  paper  review  sheets  filled  out;  however  2  of  the  9  had  no  reviewer 
sign  off  of  approval  of  actions  taken.  For  the  working  papers  that  were  signed  off  by  the 
Project  Manager  the  review  was  timely.  However,  for  the  remaining  working  papers  we 
could  not  tell  whether  supervisory  review  was  conducted  or  timely.  The  Project  Manager 
stated  that  critical  working  papers  should  have  been  reviewed  based  on  then  current 
NAVAUDSVC  guidance.  However,  we  found  that  21  of  the  59  working  papers  were 
cross-referenced  to  the  report  and  therefore  considered  critical  and  quite  often 
documented  significant  facts  and  figures  in  the  report. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy  remind  SAP  supervisors  to  sign/initial  and  date  their  review  of  working  papers  and 
that  working  papers  used  to  support  referenced  draft/final  report  statements  (specifically 
facts  and  figures)  should  be  reviewed  by  the  supervisory  personnel. 

Management  Comments.  The  NAVAUDSVC  concurred  with  the 
recommendation  and  stated  that  an  all-hands  e-mail  will  be  issued  by 
November  30,  2005,  reminding  all  personnel  of  existing  NAVAUDSVC  requirements 
related  to  documentation  of  supervision. 

Reviewer  Response.  Management  comments  are  responsive. 

Evidence  and  Audit  Documentation.  Working  papers  are  used  to  organize,  prepare, 
and  collect  relevant  documentation  and  records  during  an  audit.  GAS  7.66  requires  that 
auditors  prepare  and  maintain  audit  documentation,  and  that  the  audit  documentation 
should  contain  support  for  findings,  conclusions,  and  recommendations  before  auditors 
issue  their  report.  GAS  7.68  states  that  the  audit  documentation  forms  the  principal 
support  for  the  auditors’  report.  In  addition,  the  September  2002  NAVAUDSVC 
Handbook  Section  506.2  required  auditors  to  include  “basic  labeling  information”  such  as 
the  auditor’s  name,  date  prepared,  the  source  of  the  information  (if  the  source  was  an 
individual,  this  should  include  the  person’s  rank  or  grade;  name;  position  title;  telephone 
number;  organization;  and  date,  time,  and  place  information  was  provided),  and  the 


7 


purpose  of  the  working  paper.  NAVAUDSVC  Handbook  Section  509.1  stated  that  “All 
facts  and  mathematical  computations  in  draft  and  final  reports  and  related  working  paper 
summaries  must  be  cross-referenced  to  underlying  working  papers.” 

We  found  that  the  working  papers  generally  contained  sufficient,  competent,  and  relevant 
evidence  to  support  the  judgments  and  conclusions  in  the  reports;  however, 
documentation  could  have  been  improved  in  all  three  reports.  In  one  report,  we  found 
that  the  auditor  did  not  document  the  “date  prepared”  on  several  working  papers  used  to 
document  meetings  with  client  management.  This  is  mitigated  somewhat  by  the  fact  that 
the  working  papers  did  include  the  dates  of  the  meetings.  In  addition,  we  reviewed  more 
than  1 5  judgmentally  selected  facts  and  figures  in  the  report  and  found  that  all  were 
supported  by  information  in  the  working  papers.  However,  in  eight  of  the  sample  items, 
we  found  figures  that  were  inadequately  cross-referenced  to  supporting  documentation. 
Also,  in  several  cases,  figures  cross-referenced  to  source  documents  did  not  have 
adequate  source  information  (names  and  phone  numbers).  Further,  one  figure 
cross-referenced  to  briefing  charts  prepared  by  management  was  not  verified  against 
source  documents  in  the  working  paper  files  (this  figure  was  the  universe  of  transactions 
which  the  auditors  drew  their  sample  from). 

In  the  second  report,  though  the  facts  and  figures  in  the  report  were  verified  by  the 
independent  referencing  reviewer,  the  working  papers  supporting  numerous  facts  and 
figures  in  the  report  were  not  documented  as  reviewed  by  a  supervisor.  In  addition, 
improvements  could  be  made  in  the  independent  reference  review  process  to  ensure  that 
corrected  references  are  not  only  included  and  approved  on  the  independent  referencing 
review  comment  sheet  but  also  changed  in  the  cross-referenced  version  of  the  report.  We 
found  instances  where  the  corrected  reference  on  the  review  sheet  was  not  changed  in  the 
cross-referenced  report  resulting  in  the  risk  of  transferring  incorrect  references  to  the  final 
report. 

In  the  third  report,  we  judgmentally  selected  1 5  facts  and  figures  from  the  report  and 
determined  that  all  of  them  were  properly  supported.  However,  improvements  could  be 
made  for  cross-referencing  working  papers.  Specifically,  the  summary  working  paper 
was  not  cross-referenced  to  the  supporting  working  papers  and  a  better  job  could  have 
been  done  of  cross-referencing  numbers  in  the  individual  working  papers  and  Excel 
spreadsheets  to  the  source  documents.  The  source  documents  included  such  things  as 
various  reports  and  other  information  obtained  from  contract  files. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  all  SAP  auditors  to  comply  with  established  guidance  for  working  papers, 
documentation,  and  audit  evidence,  including  transferring  corrected  references  to  the 
cross-referenced  report. 
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Management  Comments.  The  NAVAUDSVC  concurred  with  the 
recommendation  and  stated  that  an  all-hands  e-mail  will  be  issued  by 
November  30,  2005,  reminding  auditors  of  existing  GAS  and  NAVAUDSVC  guidance 
related  to  working  papers,  documentation,  and  audit  evidence. 

Reviewer  Response.  Management  comments  are  responsive. 

Quality  Assurance.  The  June  1994  version  of  GAS  and  paragraph  3.49  of  the  June  2003 
version  required  each  organization  to  have  an  appropriate  internal  quality  control  system 
in  place.  The  November  2001  version  of  the  NAVAUDSVC  Handbook  stated  that  an 
example  of  quality  control  in  the  NAVAUDSVC  was  a  referencing  validation  performed 
by  an  independent  auditor.  We  found  that  NAVAUDSVC  had  implemented  appropriate 
internal  quality  controls  for  the  three  reports  we  reviewed;  however,  documentation  could 
have  been  improved  for  one  report.  While  an  Independent  Referencing  Review  was  done 
before  the  draft  report  was  issued  and  there  was  evidence  that  comments  that  the 
Independent  Referencer  had  were  adequately  addressed  by  the  Project  Manager, 
improvements  could  be  made  in  completing  the  referencer’ s  certification.  While  the 
referencing  certification  statement  was  signed  by  the  Independent  Referencer,  the 
statement  was  not  signed  by  the  Project  Manager  and/or  Audit  Director. 

GAS  3.50  requires  that  an  audit  organization’s  internal  quality  control  system  should 
include  procedures  for  monitoring,  on  an  ongoing  basis,  whether  the  policies  and 
procedures  related  to  the  standards  are  suitably  designed  and  are  being  effectively 
applied.  This  is  often  referred  to  as  an  internal  quality  assurance  program. 

The  NAVAUDSVC  Handbook  provides  guidance  on  the  NAVAUDSVC  Quality  Control 
Program.  The  NAVAUDSVC  Quality  Control  Program  includes  internal  quality  control 
checks  and  reviews.  As  part  of  the  NAVAUDSVC  Quality  Control  Program, 
independent,  internal  quality  control  reviews  of  selected  audits  or  segments  of  audits  will 
be  conducted  to  provide  reasonable  assurance  NAVAUDSVC  policies  and  procedures 
were  being  followed  and  are  in  accordance  with  GAS.  The  internal  quality  control 
reviews  will  be  performed  in  accordance  with  the  NAVAUDSVC  Handbook.  In 
addition,  the  quality  control  review  results  will  be  documented  and  recommendations 
tracked  through  to  final  resolution.  The  NAVAUDSVC  Special  Audits  Division  was 
responsible  for  audits  of  intelligence,  compartmented  programs  and  sensitive  activities. 

According  to  NAVAUDSVC  personnel,  audits  classified  as  requiring  special  access  are 
only  a  portion  of  the  Special  Audits  Division  workload.  Other  information,  such  as 
intelligence  or  intelligence  related  information  might  be  compartmented  or  not.  The 
audit  work  is  not  unique;  the  work  is  only  classified  at  a  higher  level  with  additional 
access  restrictions.  The  Special  Audits  Division  products  that  do  not  require  special 
access  are  always  eligible  for  internal  quality  assurance  reviews.  The  NAVAUDSVC 
Handbook  does  not  exempt  or  waive  SAP  audits  from  internal  quality  assurance  reviews. 
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However,  NAVAUDSVC  personnel  from  the  Policy  and  Oversight  Division  stated  that 
SAP  audits  have  never  been  included  as  part  of  the  internal  quality  assurance  review 
program  except  for  controls  over  things  like  continuing  professional  education  that  do  not 
require  the  reviewer  to  have  access  to  the  highly  classified  report  and/or  working  papers. 

Special  Audits  Division  products  that  do  require  special  access  should  be  subject  to 
internal  quality  assurance  reviews  due  to  challenges  in  meeting  the  other  quality 
processes  that  are  identified  in  this  report.  The  need  for  special  access  and  security 
requirements  in  the  SAP  environment  impact  on  the  extent  of  supervision, 
documentation,  and  quality  controls  compared  to  an  unclassified  environment.  In 
addition,  occurrence  of  an  external  peer  review  is  rare  within  the  SAP  audit  environment. 
Periodically  performing  internal  quality  assurance  reviews  on  SAP  audits  would  allow 
for  greater  reliability  on  the  quality  (including  accuracy)  of  these  very  important  SAP 
audit  reviews  and  the  associated  reports  that  address  areas  and  programs  that  impact 
heavily  on  our  national  security. 

We  recognize  that  the  usual  NAVAUDSVC  quality  assurance  program  may  need  to  be 
adapted  to  the  SAP  environment.  However,  having  the  internal  quality  assurance 
visibility  within  the  SAP  environment  is  critical  to  an  effective  quality  program. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  SAP  supervisors  to  sign  the  referencer  certification  statement. 

Management  Comments.  The  NAVAUDSVC  concurred  with  the 
recommendation  and  stated  that  an  all-hands  e-mail  will  be  issued  to  auditors  by 
November  30,  2005,  reminding  supervisors  to  sign  the  referencer  certification  statement. 

Reviewer  Response.  Management  comments  are  responsive. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  adapt  its  internal  quality  control  review  program,  as  needed,  for  SAP  audits. 

Management  Comments.  The  NAVAUDSVC  partially  concurred  with  the 
recommendation  and  stated  that  they  will  adapt  its  internal  quality  control  program  to 
SAP  audits  as  needed.  However,  the  NAVAUDSVC  also  stated  that  this  would  entail 
only  continuing  to  include  SAP  auditors  in  cross-cutting  internal  quality  control  reviews, 
and  not  reviewing  specific  SAP  audits.  The  NAVAUDSVC  stated  that  including  SAP 
audits  in  internal  quality  control  reviews  would  be  contrary  to  DoD  policy  because  it 
would  require  obtaining  access  to  the  SAPs  for  additional  personnel  without  the 
justification  of  additional  oversight  to  the  SAP. 

Reviewer  Response.  Management  comments  are  partially  responsive.  Based  on 
our  review  and  understanding  of  the  NAVAUDSVC  process,  they  include  the  SAP 
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auditors  in  the  internal  quality  assurance  process  up  to  a  point.  While  the  examples  they 
cited  in  their  comments  -  certifications  and  continuing  professional  education  -  are 
important,  they  do  not  include  a  review  of  critical  elements  of  the  audit  standards  related 
to  audit  planning,  evidence  and  documentation,  and  reporting.  Without  a  review  of 
project-specific  audit  work  and  documentation,  it  is  not  possible  for  NAVAUDSVC  to 
verify  that  SAP  auditors  are  complying  with  guidance  and  standards.  NAVAUDSVC 
needs  to  insure  more  than  the  “administrative  requirements  in  the  Naval  Audit  Service 
Handbook”  are  met.  We  recognize  that  the  SAP  audits  could  not  be  treated  the  same  as 
non-SAP  audits  in  the  internal  quality  assurance  program,  especially  for  the  critical 
elements  listed  above.  However,  alternative  procedures  can  be  performed  that  would 
meet  the  requirements  of  GAS  for  on-going  monitoring.  We  believe  that  NAVAUDSVC 
can  include  some  of  these  highly  classified  programs  in  its  reviews  without  necessarily 
getting  additional  people  cleared.  Using  the  Program  Director,  current  staff  who  did  not 
work  on  the  audit,  or  former  staff  with  the  requisite  clearance  are  possible  options. 

We  request  that  the  Auditor  General,  Department  of  the  Navy  reconsider  his 
decision  not  to  include  SAP  audits  in  the  NAVAUDSVC  internal  quality  control  review 
program,  and  provide  additional  comments  by  September  19,  2005. 

Other  Matter  of  Interest.  GAS  7.66  states  that  "audit  documentation  should  contain 
support  for  findings,  conclusions,  and  recommendations  before  auditors  issue  their 
report."  In  addition,  GAS  8.44  states  that  reports  "should  include  only  information, 
findings,  and  conclusions  that  are  supported  by  sufficient,  competent,  and  relevant 
evidence  in  the  audit  documentation."  The  July  2004  Draft  PCIE  Guide  required 
agencies  being  reviewed  to  either  provide  working  papers  to  the  reviewers  within 
2  working  days  of  notification  that  a  project  has  been  selected  for  review.  If  the  agency 
cannot  provide  the  working  papers  within  2  working  days,  PCIE  provides  a  form  that  the 
agency  must  use  to  either  certify  that  the  working  papers  had  not  been  changed  after  the 
final  report  was  issued  or  document  changes  that  had  been  made  since  the  report  was 
issued  and  explain  why  the  changes  were  made. 

NAVAUDSVC  personnel  agreed  during  the  planning  phase  of  the  FY  2005  round  robin 
external  peer  review  to  follow  the  July  2004  Draft  PCIE  Guide.  At  the  start  of  the  round 
robin  external  peer  review  it  was  agreed  that  the  Military  Department  audit  agencies 
would  conduct  the  external  peer  review  of  the  unclassified  and  collateral  audits  while  the 
Office  of  the  Inspector  General;  Department  of  Defense  would  conduct  the  review  of  the 
SAP  audits.  Due  to  the  unique  security  requirements  of  auditing  special  access 
information,  at  least  48  working  days  passed  between  when  we  notified  NAVAUDSVC 
of  the  audits  selected  for  review  and  when  we  obtained  physical  access  to  the  working 
papers.  However,  NAVAUDSVC  personnel  chose  not  to  certify  that  the  working  papers 
had  not  been  changed  since  reports  were  issued,  or  to  document  and  explain  any  changes 
made  to  the  working  papers  because:  (1)  the  working  papers  were  available  within  48 
hours  as  required  by  the  draft  PCIE  guide  just  not  to  the  peer  review  team;  (2)  the 
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selected  audits  were  several  years  old  and  the  working  papers  were  all  hard  copy;  and  (3) 
none  of  the  NAVAUDSVC  staff  who  had  worked  on  the  audits  were  still  in  the  Special 
Audits  Division.  Therefore,  NAVAUDSVC  personnel  certified  that  the  working  papers 
had  not  been  changed  since  the  date  that  they  provided  us  the  universe  of  SAP  audit 
reports  issued  during  FYs  2002,  2003,  and  2004.  However,  the  working  papers  were  not 
technically  available  given  that  NAVAUDSVC  could  not  provide  us  with  access  to  the 
working  papers  until  the  appropriate  security  procedures  were  taken  care  of. 

NAVAUDSVC  personnel  stated  that  NAVAUDSVC  policy  at  the  time  of  our  review  was  to 
include  any  post-audit  work  (including  responses  to  final  report  and  follow-up  of 
recommendations)  in  the  working  papers  of  the  audit.  As  a  result  of  concerns  about  post¬ 
report  issuance  changes  to  audit  documentation  and  notification  of  peer  review  as  part  of 
the  round  robin  performed  by  the  Army  Audit  Agency,  NAVAUDSVC  is  instituting  a 
new  procedure  for  compiling  all  post-audit  documentation  in  a  file  separate  from  the  rest 
of  the  audit  documentation.  This  action  should  safeguard  audit  documentation  used  to 
support  findings,  conclusions,  and  recommendations  after  final  report  issuance.  Though 
manual  working  papers  are  used  in  the  SAP  environment,  NAVAUDSVC  should 
incorporate  the  new  procedure  within  the  SAP  audit  environment  to  reduce  the  potential 
that  audit  documentation  used  to  support  findings,  conclusions,  and  recommendations  in 
the  final  report  can  be  inappropriately  changed. 

Recommendation.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  ensure  that  the  new  procedure  on  post-audit  documentation  be  incorporated  into 
the  SAP  audit  process. 

Management  Comments.  The  NAVAUDSVC  concurred  and  stated  that  they 
will  develop  and  implement  post-audit  procedures  for  SAP  audit  documentation  by 
November  30,  2005. 

Reviewer  Response.  Management  comments  are  responsive. 
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Appendix  C.  Management  Comments 


DEPARTMENT  OF  THE  NAVY 

NAVAL  AUDIT  SERVICE 
1006  BEATTY  PLACE  SE 
WASHINGTON  NAVY  YARD,  DC  20374-5005 


24  Aug  2005 

MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  ASSISTANT  INSPECTOR 
GENERAL  (AUDIT  POLICY  AND  OVERSIGHT) 

Subj:  COMMENTS  ON  DRAFT  REPORT  ON  QUALITY  CONTROL  REVIEW  OF 
NAVAL  AUDIT  SERVICE’S  SPECIAL  ACCESS  PROGRAM  AUDITS 

Ref:  (a)  Memorandum  for  Auditor  General,  Department  of  the  Navy  -  Quality  Control 
Review  of  Naval  Audit  Service’s  Special  Access  Program  Audits 
(Project  No.  D2005-DIPOAI-0098) 

Enel:  (1)  Department  of  Defense  Assistant  Inspector  General  (Audit  Policy  and  Oversight) 
Recommendations  and  Naval  Audit  Service  Comments 

1 .  We  have  reviewed  the  report  on  the  Naval  Audit  Service’s  Special  Access  Program  Audits 
(reference  a),  and  are  pleased  that  Department  of  Defense  Inspector  General  has  provided  us 
with  an  unqualified  opinion  on  our  system  of  quality  control  for  the  year  ended  30  September 
2004.  Enclosure  (1)  contains  our  overall  comments  on  the  report  and  a  response  to  each  of  the 
recommendations. 

2.  If  you  have  any  questions,  or  would  like  additional  information,  please  contact  Vicki 
McAdams,  Deputy  Director  of  Policy  and  Oversight,  at  Vicki.mcadams@navv.mil  or 
(202)  433-5685;  or  me  at  ionathan.kleinwaks@navv.mil  or  (202)  433-5854. 


— /> 

JONATHAN  KLEINWAKS 
Director,  Policy  and  Oversight 
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Final  Report 
Reference 


Revised 
Report  at 
p.  5 


Department  of  Defense  Assistant  Inspector  General  (Audit 
Policy  and  Oversight)  Recommendations  and  Naval  Audit 
Service  Comments 


Overall  Comment 

The  Naval  Audit  Service  (NAVAUDSVC)  is  pleased  to  receive  an  unqualified 
opinion  on  its  quality  control  system  for  special  access  program  (SAP)  audits. 

We  have  evaluated  the  report  and  concur  with  all  recommendations.  Our  planned 
actions  are  described  later  in  this  memorandum. 

We  do,  however,  have  a  concern  with  the  draft  report  comments  pertaining  to  the 
unqualified  opinion.  Specifically,  we  disagree  with  the  Appendix  B  statement 
that  says  “We  are  issuing  an  unqualified  opinion  on  this  external  peer  review 
because  this  is  the  first  external  peer  review  of  NAVAUDSVC  audits  requiring 
special  access....”  We  believe  this  statement  should  be  removed  because  it 
implies  that  if  this  had  not  been  the  first  external  peer  review  of  our  SAP  audits,  a 
less  than  unqualified  opinion  may  have  been  issued.  In  fact,  this  is  not  the  case. 
The  Department  of  Defense  Inspector  General  (DoDIG)  peer  review  team 
concluded  and  reported  that  the  NAVAUDSVC  “internal  quality  control  system 
was  operating  effectively  to  provide  reasonable  assurance  that  SAP  audit 
personnel  were  following  established  policies,  procedures,  and  applicable  auditing 
standards,”  and  that  the  concerns  “identified  during  the  review  were  not 
cumulatively  significant  to  reports’  findings,  conclusions,  or  recommendations.” 
Clearly,  as  the  peer  review  found,  the  unqualified  opinion  has  nothing  to  do  with 
the  newness  of  the  SAP  peer  review. 

DoDIG  Recommendations  and  NAVAUDSVC  Responses 

Recommendation  1.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  all  SAP  managers  to  document  their  approval  for  all  original  and 
updated  audit  programs  (including  the  cross-referenced  version),  ensure  that  the 
audit  programs  include  steps  for  verifying  computer  data  obtained  during  the 
audit,  ensure  the  audit  program  is  adequately  cross-referenced  to  working 
papers,  and  complete  the  initial  and  dale  column  if  such  a  column  is  included  in 
the  audit  program. 

NAVAUDSVC  Response.  Concur.  An  All-Hands  email  will  be  issued 
reminding  all  personnel  of  the  Naval  Audit  Service  Handbook 
requirements  pertaining  to  audit  programs,  and  the  requirement  to  evaluate 
computer-based  products  to  determine  risks.  Action  will  be  completed  by 
30  November  2005. 

Recommendation  2.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy  remind  SAP  supervisors  to  sign/initial  and  date  their  review  of  working 
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Page  1  of  3 


14 


papers  and  that  working  papers  used  to  support  referenced  draft/final  report 
statements  (specifically  facts  and  figures)  should  be  reviewed  by  the  supervisory 
personnel. 

NAVAUDSVC  Response.  Concur.  An  All-Hands  email  will  be  issued 
reminding  all  personnel  of  this  Naval  Audit  Service  Handbook 
requirement.  Action  will  be  completed  by  30  November  2005. 

Recommendation  3.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  all  SAP  auditors  to  comply  with  established  guidance  for  working 
papers,  documentation  and  audit  evidence,  including  transferring  corrected 
references  to  the  cross-reference  report. 

NAVAUDSVC  Response.  Concur.  An  All-Hands  email  will  be  issued 
reminding  auditors  of  the  Government  Auditing  Standards  and 
NAVAUDSVC  Handbook  policy  for  working  papers,  documentation,  and 
audit  evidence.  Action  will  be  completed  by  30  November  2005. 

Recommendation  4.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  remind  SAP  supervisors  to  sign  the  referencer  certification  statement. 

NAVAUDSVC  Response.  Concur.  An  All-Hands  email  will  be  issued 
reminding  supervisor’s  to  sign  the  referencer  certification  statement. 
Action  will  be  completed  by  30  November  2005. 

Recommendation  5.  We  recommend  that  the  Auditor  General,  Department  of  the 
Navy,  adapt  its  internal  quality  control  review  program,  as  needed,  for  SAP 
audits. 

NAVAUDSVC  Response.  Concur.  The  NAVAUDSVC  will  adapt  its 
internal  quality  control  review  program  to  SAP  audits  as  needed. 

However,  it  should  be  noted  that,  at  this  time,  we  believe  our  current 
internal  quality  control  program  is  sufficient  for  SAP  audits.  Specifically, 
SAP  auditors  are  routinely  included  in  NAVAUDSVC  quality  control 
reviews  because  our  internal  quality  control  review  program  routinely 
conducts  crosscutting  quality  control  reviews  that  cover  all  audits.  These 
reviews  have  not  and  will  not  exclude  SAP  auditors.  For  example,  we 
recently  conducted  a  review  to  determine  whether  the  NAVAUDSVC 
complied  with  continuing  professional  education  requirements  relative  to 
the  auditor  competency  standard  as  defined  by  Generally  Accepted 
Government  Auditing  Standards  (GAGAS),  and  we  currently  have  a 
quality  control  review  ongoing  to  determine  whether  employee  degree  and 
certification  data  is  accurate.  Personnel  responsible  for  conducting  SAP 
reviews  have  been  and  will  be  included  in  these  quality  control  reviews. 

We  believe  this  approach  of  including  SAP  audit  staff  in  quality  control 
reviews  makes  the  most  sense,  as  it  is  consistent  with  national  security 
interests.  Conducting  routine  quality  control  reviews  of  SAP  program 
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audits  would  be  contrary  to  DoD  Directive  0-55205.7,  which  states, 
“Granting  access  to  a  DoD  SAP  shall  be  based  solely  upon  a 
determination  that  the  individual  has  a  valid  need-to-know...and  will 
clearly  and  materially  contribute  to  the  execution  or  oversight  of  the 
program.” 

Neither  our  internal  quality  control  reviews  nor  this  external  peer  review 
have  identified  any  issues  that  “clearly  and  materially  contribute  to  the 
execution  or  oversight  of  the  program.”  Specifically,  quality  control 
reviews  have  typically  identified  issues  such  as  the  inadequate 
documentation  of  supervision  and  lack  of  compliance  with  administrative 
requirements  in  the  Naval  Audit  Service  Handbook,  and  poor 
recordkeeping  procedures.  While  these  are  important  issues,  we  do  not 
need  to  and  should  not  provide  access  to  the  some  of  the  nation’s  most 
highly  classified  programs  to  identify  and  resolve  such  issues. 

Additionally,  when  we  implement  process  improvements  to  address 
quality  control  findings,  we  routinely  apply  them  to  all  of  our  audits, 
including  the  SAP  audits.  This  is  possible  because,  as  the  peer  review 
report  notes,  SAP  “audit  work  is  not  unique.”  In  fact,  the  SAP  audits 
reviewed  in  this  peer  review  covered  areas  similar  to  other  audits.  Thus, 
all  of  our  audits,  including  the  SAP  audits,  are  able  to  benefit  from  internal 
quality  control  reviews,  without  us  needing  to  provide  unnecessary  access 
to  the  SAP  programs. 

In  light  of  the  nature  of  the  findings  presented  in  both  our  internal  quality 
control  reviews  and  the  external  peer  review,  at  this  time  there  is  no 
justification  for  deviating  from  our  current  approach  of  including  SAP 
auditors  in  crosscutting  internal  quality  control  reviews.  Action  is 
considered  complete. 

Recommendation  6.  We  recommend  that  the  Auditor  General,  Department  of  the 

Navy,  ensure  that  the  new  procedure  on  post-audit  documentation  be 

incorporated  into  the  SAP  process. 

NAVAUDSVC  Response.  Concur.  We  have  recently  implemented 
automated  post-audit  documentation  procedures  for  non-SAP  audits. 
Because  SAP  work  papers  are  manual  and  subject  to  special  security  and 
storage  rules,  different  post-audit  documentation  procedures  are  being 
developed  and  implemented  for  these  special  work  papers.  Action  will  be 
completed  by  30  November  2005. 
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